Conversation
The chunker could cut ranges but nothing copied them; the applier needs a per-key uncut/in-flight/landed answer that only the copier can give. Promotes the in-transaction lock check to dbconn so copier and builder share one LK-1 confirmation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
copier.Copier, the parallel chunked copy from a proven source into its built shadow, and promotes the in-transaction table-lock confirmation intopkg/dbconnso the copier and the shadow builder share it.Why
The chunker (previous PR in this stack) cuts consecutive key ranges but nothing copied them. The applier that follows needs more than a watermark: with several workers, chunks land out of order, so a captured change must be judged per key as uncut (discard), in flight (defer), or landed (apply). Only the copier knows which chunks are in flight, so it owns that answer.
The shadow builder already re-asserted the lock from inside its own transaction; the copier needs the identical check on every chunk connection. One implementation in
dbconnkeeps LK-1 enforced in one place.What
pkg/dbconn:(*TableLockSession).Confirm(ctx, conn)— nobody holds the lock →ErrInvariantViolationwrapping the newErrTableLockNotHeld; another backend holds it →*TableLockHeldErrornaming it.schemachange.confirmTableLocknow maps those onto its existingCauseLockUnconfirmed/CauseLockHeldElsewhererefusals (existing tests unchanged).pkg/copier:Shadowinterface (schema, source, shadow, both OIDs, copy columns) — the shapeschemachange.BuiltShadowsatisfies, so the copier stays importable by the builder.NewCopierrefuses a shadow that is not the target's (ST-6) and a lock session that is missing, for another table, or already lost (LK-1).Copier.Run(ctx, pool): N workers (default 4) under the lock session'sBindcontext. Each chunk runs in its own transaction:SET LOCAL lock_timeout/statement_timeout,SET LOCAL ROLE owner,lock.Confirm, re-resolve both relation OIDs against the proof, then one frozen statementINSERT INTO shadow (cols) SELECT cols FROM source WHERE pk BETWEEN $1::bigint AND $2::bigint ON CONFLICT (pk) DO NOTHING. Elapsed time from an injectedprogress.ClockfeedsChunker.Feedback(D12).Position: a chunk is registered in flight before its transaction begins and removed only after commit or rollback; the watermark advances over the contiguous landed prefix;Position.Classify(key)returnsKeyUncut/KeyInFlight/KeyLanded— the CO-4 rule the applier will call.Runreturns only after every worker has exited (LK-3), then fails closed if anything is still in flight or the key space is not covered.SAFETY.mdcopier row,invariants.mdCO-4 / LK-1 / LK-3 Enforced today, design package map (dbconn,copier) and D12,architecture.md.Tests (real PostgreSQL, PG 14/16/18): whole-table copy with 3 workers and 700-row chunks converges via
testutil.AssertConverged; a pre-existing shadow row is never overwritten; resume from a watermark copies only the keys above it; cancellation with one chunk pinned mid-insert by an uncommitted shadow row returnscontext.Canceledwith nothing in flight, every key ≤ watermark present, and a resumed copier converges; lock loss mid-copy returnsErrInvariantViolationwrapping the session's loss; a dropped-and-recreated shadow or source is refused (ST-6) with zero rows written; a gone or rival-held lock is refused with zero rows written; the copy SQL is frozen as an exact string (TM-2); pure ledger tests for out-of-order landing, release-without-landing, and resume.Before / after
References
kiran01bm/cs5-copier); next PR adds the progress fillers.docs/invariants.mdCO-4, LK-1, LK-3;docs/copy-and-swap-design.mdD12 and the package map.🤖 Drafted with Amp (Claude Opus 4.6); reviewed and edited by the author.